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ABSTRACT 


Automation and robotics have played important roles in space activities for many 
years, most notably in planetary exploration and Space Shuttle operations. There are 
several major robotic systems being designed and developed for the Space Station. One 
such system is the Extravehicular Activity Retriever (EVAR). The EVAR will be an 
intelligent free-flying robot which will have the ability to locate, track, maneuver to, and 
retrieve objects which have been detached from the Space Station. Most notably, the 
EVAR is being designed so that it will be able to retrieve crew members who have become 
stranded during extravehicular activity. The EVAR is undergoing a series of ground-based 
demonstrations in the precision air bearing floor facility located at the NASA Johnson 
Space Center. These tests serve as a means of providing proof of concept as well as 
demonstrating system level performance of the EVAR hardware and software. 

Space-based robotics systems are presented with many technical problems which can 
be avoided in robots designed for terrestrial settings. Due to the near absense of gravity in 
space, the manipulation of objects is quite different from what is normally experienced on 
earth. In contrast to the structured environments in which factory robots operate, 
space-based robots must deal with environments containing a high degree of uncertainty. 
'Die harsh nature of lighting in space creates visual perception problems as well. The end 
result is that although many robots have been developed which perform adequately on 
earth, much technology remains to be developed to support intelligent space-based robots 
such as the EVAR. 

This study considers one such area of technology, the development of an approach to 
the visual perception of object surface information using laser range data in support of 
robotic grasping. This is a very important problem area in that a robot such as the EVAR 
must be able to formulate a grasping strategy on the basis of its knowledge of the surface 
structure of the object This paper presents a description of the problem domain and 
formulates an algorithm which derives an object surface description adequate to support 
robotic grasping. The algorithm is based upon concepts of differential geometry namely, 
gaussian and mean curvature. 
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INTRODUCTION 


It is the purpose of this paper to formulate an algorithm for the calculation of surface 
shape and structure from laser range data. This algorithm was developed with the intent of 
using it to assist in the closed loop control of the grasping operation performed by the 
Extravehicular Activity Retriever (EVAR). In order to perform the grasping operation a 
robot such as the EVAR must have knowledge of the structure of the surface of the object 
to be grasped. Using such information, the robot can plan the reach of its manipulator to 
the object and can preshape its hand in such a way so as to accommodate the object and 
ensure a stable grasp. The algorithm calculates information about the surface structure and 
is based upon concepts from differential geometry, namely the gaussian and mean 
curvatures of a surface. The approach to be presented is based in part upon previous 
works by other investigators [1], [2]. 

This report is based upon preliminary results from a study initiated this summer at the 
Johnson Space Center. In addition to the presentation of the algorithm for curvature-based 
surface calculation, this report also describes some of the key criteria relating to the 
determination of grasping strategies. It also presents a framework which allows a robot to 
store essential surface characterization and grasp configuration information in a 
computerized library. Under such a framework, a robot can make use of previously 
generated knowledge about objects and hand configurations to simplify the complexity of 
grasp planning. 


EXTRAVEHICULAR ACTIVITY RETRIEVER 

The use of robots in hazardous environments such as space is rapidly expanding. 
Numerous studies and symposia have been conducted on this very important topic [3], 

The EVAR is a voice-supervised, intelligent, free-flying robot which is currently being 
designed, developed, and demonstrated in a ground-based laboratory facility at the Johnson 
Space Center [4]. When operational, the EVAR will be used to retrieve objects such as 
construction materials, equipment, and tools which have been accidentally separated from 
the Space Station. The EVAR will be in a stand-by mode whenever astronauts perform 
extravehicular activity (EVA). If an astronaut becomes detached from the Station, the 
EVAR will be summoned to locate, fly to, grapple and return the astronaut to the Station. 
The EVAR is being designed so that it can accommodate an unconscious or uncooperative 
astronaut. 

The EVAR consists of six integrated subsystems: the Manned Maneuvering Unit 
(MMU), two robotic arms with grippers, a video and tracking system, a 3D laser imaging 
system, an onboard computer system, and a data and control network. The MMU provides 
the EVAR with the ability to fly about in a manner similar to that of an astronaut during 
EVA. The two robotic arms are anthropomorphic in nature. One of the grippers is a 
dexterous hand while the other is a parallel jaw gripper. The EVAR video and tracking 
system employs two black and white cameras, a tracking system, and a monitor to allow 
for backup supervisory control of EVAR operations by personnel at a remote location. One 
camera enables target acquisition and tracking, while the second can be used to assist 
during target grappling. The 3D laser imaging system is an Ode tics 3D laser radar ranger. 
This subsystem, often called a 3D mapper, consists of a scan unit, an electronics unit, and 
a power supply. The 3D mapper provides direct digital range measurement and is capable 
of scanning a 60 degree horizontal by 60 degree vertical area with a 128X128 raster scan. 
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The frame rate of the system is every .835 seconds. The onboard computer system 
consists of several transputers and other processors. The data and control network enables 
the routing of communication and control information among the various subsystems of 
the EVAR. A three phase ground demonstration program for the EVAR has been planned 
and scheduled. The first phase of the program was successfully completed this past year. 
The second milestone demonstration will be conducted this fall at the precision air-bearing 
floor facility at NASA Johnson Space Center. 

ROBOTIC GRASPING 

Robot manipulation is a complex process that consists of several steps: sensing, task 
planning, trajectory planning, grasping, and path following. With respect to grasping, 
there are three principal grasp selection criteria which are of major concern. They are 
described briefly below. 

Safety. The criteria of safety involves the issue of damage control as applied to three 
areas: (i) the target object must be safe in the sense that there should not be any damage to 
the object during the grasp operation; (ii) the robot must avoid damaging itself as it 
performs the operation; and (iii) the task environment should remain damage-free during 
tiie operation. Whereas the EVAR is being developed to retrieve stranded crew members 
and possibly very expensive Space Station construction tools and materials, requirement (i) 
is an overriding concern in the development of grasping strategies for the EVAR. In 
addition, the EVAR itself will be an expensive piece of equipment, therefore elevating the 
importance of requirement (ii). 

Reachability. This criteria centers on the ability of the hand to reach a particular 
target In other words, there must be a sufficiently large workspace to accommodate the 
movement of the hand during the operation. This requires that the hand be able to reach the 
object at some initial hand configuration and, with the object in hand, to follow a 
collision-free path to a final location. In addition, the workspace of the hand's fingers must 
be large enough relative to the size of the object for the fingers to secure the proper grasp. 

Stability. The grasp must be stable in the presence of external forces which could be 
exerted on the grasped object of the hand during the manipulation process. By stable, we 
simply mean that a grasp does not permit the target to move or slip with respect to the hand. 

It is clear that the above grasp criteria are closely related to the characteristics of the 
targets and the environments which they occupy. In fact, they form the bases upon which 
grasp configurations and operation strategies are determined. In the following section, the 
characteristics of targets which are most critical to the grasping operation are presented. 

CHARACTERISTICS OF OBJECTS 

Objects can be characterized by three different types of attributes (geometric, 
physical, and mechanical) [5]. Geometric attributes include the size of an object, the shape 
of the object, and the shape of the contact surface. With regard to grasping, the size of the 
object is most important in determining the accessibility of the object The shape of the 
object as well as that of its contact surface influence grasp stability and hand preshaping. 
Prior to selecting a grasp, a list of candidate grasp configurations can be extracted from a 
library. Candidate grasp configurations can then be evaluated on the basis of the geometric 
attributes of the object The set of candidate grasp configurations could then be narrowed 
by removing those grasp configurations that would lead to unstable grasps. Finally, the 
physical and mechanical characteristics of the object could be taken into consideration. 

Physical attributes include the mass, mass distribution, and inertia of the objects. The 
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center of mass affects the stability of the grasp configurations. The physical attributes of 
the target are difficult, if not impossible, to extract using solely non-contact methods of 
sensing, such as imaging or laser ranging. This being the case, it would be advantageous 
to develop a library containing the physical properties of objects likely to be retrieved by the 
EVAR [6]. By classifying the target, the EVAR could then retrieve its physical attributes 
and make use of these in the formulation of a grasp strategy. 

Target objects can be grouped into rive categories based upon mechanical attributes. 
These categories are rigid, brittle, elastic, flexibile, and slippery objects. Rigid objects 
exhibit a high degree of stiffness which makes them unlikely to deform during grasping. 
Unlike rigid objects, brittle objects have low stiffness. Potentially, objects of this type can 
be very fragile and thus easily damaged. Grasping brittle objects requires accurate 
information concerning the shape and size of contact areas in that they can be easily 
damaged. Elastic objects are characterized on the basis of behavior of their surfaces about 
the point of contact by a manipulator. When a force is exerted on an elastic object, 
deformation of the surface at the point of contact of the manipulator may appear. When the 
force is removed, the deformation disappears, restoring the object to its original form. 
Elastic objects are typically the easiest objects to grasp. Unlike elastic objects, flexible 
objects have low stiffness. As a result, deformation which result from the application of a 
force remain subsequent to the removal of the force. Normally, detailed shape information 
about the target is not as important for grasping elastic objects. Large contact areas are 
essential for grasping flexible objects. Slippery objects are perhaps the most difficult to 
grasp and handle. Whereas the coefficient of friction at the contact area is very low, the 
stability of the grasp can be diminished. 

It is important to note that the mechanical attributes of objects should be considered in 
conjunction with geometric and physical attributes in the determination of grasp strategies. 
Such design ensures that any drawback in one aspect can be compensated by advantages 
offered by the others. 

VISUAL PERCEPTION OF RANGE IMAGING 

In a broad sense, visual perception of range images can be viewed as the process of 
interpreting measurements made using any of a variety of range sensors. Because the 
success of automated devices depends critically on systems with the ability to sense and 
understand the environment, range image perception has received the attention of many 
investigators. Automatic systems have been designed to work in tightly structured 
environments such as assembly lines in factories. Unfortunately, much of the technology 
associated with systems of this type is not directly transferrable to the EVAR in that the 
EVAR must have the ability to cope with uncertainities in its environment 

The EVAR is equipped with a laser ranging device which is capable of determining a 
range image of the environment surrounding the robot A range image is merely a large 
collection of distance measurements from a known reference coordinate system to surface 
points on objects within a scene. If the distance measurements in a range image are listed 
relative to three orthogonal coordinate axes, the range image is said to be in xyz form. If 
the distance measurements indicate range along 3D direction vectors indexed by two 
integers ( i, j ), the range image is said to be in r jj form. Any image in ry form can be 

converted directly to xyz form, but the converse is not true [7]. The term image is used 
because any ry range image can be displayed on a video monitor, and it is identical in form 

to a digitized video image from a television camera. In addition the term visual perception 
is often associated with the extraction of information from a range image for the same 
reason. 
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Range images can be a source of valuable information for an automated machine. 

The EVAR can use range image information to determine the location and orientation of 
objects in its environment This information can be used to plan the EVAR's movements 
and actions. Surface fitting techniques make use of range data to produce a geometric 
description of objects. In this report we are interested in determining the underlying 
surface structure of targets which the EVAR will be responsible for grasping. An approach 
to extracting the surface structure of a target is critical to the grasp operation because 
manipulators grasp surfaces. The following section presents an approach to surface fitting 
which is based upon curvature concepts taken from the field of differential geometry. 

SURFACE FITTING 

The surface fitting problem can be stated as follows: given three m x n arrays of x, y, 
and z coordinates, we would like to determine a surface that approximates the data in the 
least-squares sense and that is smooth. In a mathematical sense, smoothness is defined as 
the twice-differentiability of the surface at all points. Fitting a surface to an L x L window 
of data is equivalent to computing a surface fit for a roughly rectangular grid of data values. 
Intuitively, a roughly rectangular grid is one that has been obtained from a rectangular 
mesh that has been deformed to fit the surface. We may represent the surface in parametric 
form by the three equations 

x = f(s,t), 

y = g(s,t), 

z = h(s,t), 

where s and t are the parameters, and the functions f, g, and h are tensor products of 
splines in tension. The fitting can be viewed as a mapping from 2D space to 3D space. 

The 2D space is characterized by the parameters s and t and the 3D space is the standard 
cartesian space. 

Principal curvatures at a point on a surface indicate how fast the surface is pulling 
away from its tangent plane at that point. Principal curvatures can be computed by 
estimating partial derivative infomation for the surface. The principal curvatures will 
achieve a local maximum in the area surrounding a discontinuity in the surface, for example 
an edge in the object By thresholding the values of the principal curvatures, we can 
declare the edge points in a range image. 

When a plane passing through the normal to the surface at a point P is rotated about 
this normal, the radius of curvature changes and will be a maximum distance rj fen- a 

definite normal section sj and a minimum 13 for another normal section S 2 - The 

reciprocals 1 <j = \fv\ and ~ U*2 316 called the principal curvatures; the directions of the 

tangents to si and S 2 at P are called the principal directions of the surface at P. Gaussian 

curvature at point P is defined as the product of the two principal curvatures. It can be 
proven that the gaussian curvature depends only upon the coefficients of the first 
fundamental form of a surface and their derivatives. The coefficients of the second 
fundamental form share the previously stated property. The first and second fundamental 
coefficients also determine the surface uniquely up to a rigid body transformation. As a 
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result, the gaussian curvature is said to be an intrinsic property of the surface. The 
principal curvatures at a point on a surface can be computed in terms of the parameters s 
and t using the following approach. Let X(s,t) represent the surface 

X(s,t) = [ x(s,t), y(s,t), z(s,t) ] T . 

Here, bold face notation is used to denote a vector quantity. If we let partial differentiation 
be represented by subscripts, the differential element dX is a vector given by the 
relationship 

dX = X s ds + X t dt, 
where 

X s = dX/ds = [ x s , y s , z s ] T , 


X t = dX/dt = [ x t , y t , z t ] T . 


If we take the scalar product of dX with itself we obtain a relationship which is known as 
the First Fundamental Form of the surface, I, which is given below: 

I = dX • dX = ( X s ds + X t dt ) • ( X s ds + X t dt ) 

= Eds 2 + 2Fdsdt + Gdt 2 > 

where the First Fundamental Coefficients (E, F, and G) are given by: 

E = I X s I 2 = x s 2 + y s 2 + z s 2 , 

F = X s • X t = x s x t + y s y t + z s z u 
G = I X t I 2 = x t 2 + y t 2 + z^. 

The unit normal is then given by 
N = (X s xX t )/IX s xX t l. 

The differential of the unit normal is 
dN = N s ds + N t dt. 

If we take the negative of the scalar product of the differentials of the surface and the unit 
normal, we obtain the Second Fundamental Form for the Surface, 

n = -dX • dN = - (X s ds + X t dt) • (N s ds + N t dt) = 

= L ds 2 + 2 M ds dt + N dt 2 * 
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where the Second Fundamental Coefficients (L, M, and N) are given by: 

L = -X s • N s , 

M = -1/2 ( X s . N t + X t . N s ), 

N = - X t • N t . 

The gaussian curvature K at any point on the surface is defined as the product of the two 
principal curvatures T<j and ^ and can be expressed in terms of the First and Second 
Fundamental Coefficients as 


K = 'k 1 l < 2 = (LN-M2)/(EG-F 2 ). 


The mean curvature H is given by the average of the two principal curvatures. It may be 
expressed in terms of the First and Second Fundamental Coefficients as well. 


H = ( LG - 2MF + NE ) / ( 2( EG-F 2 ) ). 
The principal curvatures are given by, 


kj = H - SQRT( H 2 - K ), 


k 2 = H + SQRT(H 2 - K ). 

The above formulas can be used to compute the gaussian and mean curvatures. Let us 
assume that the gaussian and mean curvatures for a particular point referenced by the 

indices ( i, j ) in a range image have been calculated and are stored in arrays K( i, j ) and 

H( i, j ) respectively. The points on a surface can then be classified according to the signs 
of these quantities: positive, negative, or zero. It turns out that there are wily eight possible 
outcomes for surface characterization based upon the signs of the gaussian and mean 
curvatures. We introduce notation for two functions (modified signum function and the 
surface classification function). The modified signum function is a mapping of a scalar 
argument into one of three values ( 1 , 0 , or -1). For positive e, it is defined as: 

sgn e (y) = 1 if y > e 
0 if I y I < e 
-1 if y <-e . 
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We take advantage of the clarity afforded by use of the modified signum function to 
introduce the surface classification function 


T( i, j ) = l + 3(l + sgn ei (H(i,j))) + (l-sgn e 2 (K(i,j)). 


The positive scalars el ande2 are chosen to be close to zero. They enable the mapping to 
zero of a number within the limits of the numerical accuracy of the machine and the 
algorithm. Figure la gives the mapping of the values of the modified sigpum function 
applied to the gaussian and mean curvatures. Figure lb gives a pictorial interpretation of 
die eight possible surface classifications afforded by the function T( i, j ). 

CURVATURE-BASED SURFACE CALCULATION ALGORITHM 

This section presents an algorithm to compute the object description in terms of jump 
boundaries, internal edges, and regions homogeneous in the sign of their gaussian and 
mean curvatures. The algorithm can be summarize as a sequence of the following steps: 

• Divide the range image into overlapping windows. 

• Detect jump boundaries and fit patches to windows of data not containing jump 
boundaries. 

• Compute the principal curvatures and extract edge points. 

• Classify each nonedge point in a patch as one of the eight possible surface classifications 
presented previously. 

• Group all points of the same type in a patch and its neighboring patches into a region. 

The details of each step in the algorithm are as follows. In the first step, the division of the 
input arrays of ( x, y, z ) coordinates into L x L windows should be overlapped to ensure 
that an internal edge is always contained in a patch. 

In the second step, the relative degree of scatter exhibited by the data is reflected in 
the standard deviation of the euclidean distance between adjacent data points within the L x 
L window. In the vicinity of a jump boundary of an object, the standard deviation of these 
euclidean distances will be relatively high. Jump boundaries can be detected by means of a 
threshold test. After the jump boundaries have been detected, smooth patches based upon 
2D B-splines can be fitted to the data in the window not containing a jump discontinuity 
[8]. In the presence of a jump boundary, the window size can be modified to ensure that 
the window excludes the jump boundary. In order to detect fine detail in the scene it is 
important that the size of the objects be much larger than the size of the L x L window. 

In the third step of the algorithm, we attempt to detect all those points that belong to 
or fall on internal edges of the object. As was mentioned previously, in the vicinity of an 
edge, the values for die principal curvatures will be high and in fact achieve a local 
maximum . As a result, we first determine all points exhibiting principal curvatures above 
threshold. Due to the presence of noise in the data and an inappropriate choice of 
thresholds, clusters of edge points may appear in the vicinity of a true edge position. To 
eliminate these clusters of computed edge points, a suppression of nonmaxima is applied at 
every edge point. That is, we only declare an edge to be present at points where the 
curvature is a local maximum. Nonmaxima suppression is applied in a direction 
perpendicular to the edge directions which is the direction associatated with the maximum 
absolute principal curvature. 

The next step is to group object points into homogeneous regions. All points in a 
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Figure la. Classification of surfaces 
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Figure lb. Surface types 
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patch are classified into one of the types indicated in Figure lb. All points of the same type 
are grouped together into a larger region. Ther merging of points does not require explicit 
fitting of a new surface to this homogeneous region since the surfaces are chosen to be 
smooth. Each homogeneous region is assigned a label depicting its type. Internal edges 
may occur within regions. The extent of regions can be delineated by jump boundaries, 
internal edges, and curvature edges, which we define as places where there is a change in 
curvature-based classification. 

The object representation in terms of regions and curvature-based properties , as 
described above, have the following advantages: 

• it is invariant under transformation of independent parameters of the surface. 

• it is invariant to rigid body transformation and is therefore independent of viewpoint 

EVAR GRASPING 

Operational constraints of the EVAR Demonstration Program demand that grasping 
be performed in a real-time, closed-loop manner. Under Phase II of the program, both 
video and range data will be available for the demonstration. In addition, the target will be 
a stationary object It is assumed that the EVAR hand will be in close proximity (within a 
foot) of the target prior to grasping. 

Probably the single most important property of the human hand is its ability to grasp a 
wide range of objects. The incorporation of a hand on the EVAR will enable the EVAR to 
grasp a wide variety of objects as well. Grasping involves the coordination of many 
degrees of freedom. In an automated system, we wish to reduce the number of degrees of 
freedom associated with robotic grasping. One approach to achieve this goal is based upon 
the notion of classification of grasp configurations. By dealing with grasp configurations 
in a packaged format through classification, one can simplify the amount of planning 
associated with the grasping operation [9]. This approach is based upon the abilty of the 
automated system to recognize surface structures based upon features derived from image 
and range measurements. Features could be constructed using the curvature-based surface 
representation calculation algorithm to describe the underlying suface structure of the object 
to be grasped. These features could be matched against those stored in a surface/grasp 
library [10]. This library would contain grasp configuration which would relate to the 
particular surface structure. In addition, the library would contain the physical and 
mechanical attributes of the associated object. Suppose the EVAR is presented with a 
surface structure similar to one previously encountered and successfully grasped. Under 
such circumstances, the EVAR can make use of information previously generated 
concerning grasping to sieze the target This information would consist of hand preshape 
geometry, grasp approach, and other relevant information. 

If die surface cannot be classified by matching its surface features with those 
contained in the surface/grasp library, it will be necessary to formulate a new grasp 
strategy. In that the hand must grasp surfaces, the information generated using the 
algorithm would be useful in the development of a new grasp configuration. If a new 
configuration is developed and successfully used to produce a stable grasp, then it is stored 
along with the underlying target surface structure in the surface/grasp library. Needless to 
say, the surface/grasp library will be dynamic. As new objects are successfully grasped, 
the library will expand. The overall approach is illustrated in Figure 2. 

SUMMARY 

This paper has presented an algorithm for the calculation of curvature-based surface 
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Figure 2. Framework for grasp operation. 
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characteristics for objects using range data. At the present time, this algorithm is being 
implemented in the programming language C. Although the approach to surface 
description based upon curvature concepts taken from differential geometry appears 
promising, it would be presumptuous to leave the reader with the impression that it will 
provide all the information necessary for secure and stable grasping for the EVAR. It is 
essential to integrate this prototype software, when available, in a real-time systems level 
demonstration in order to fully evaluate its merits. Such a demonstration would bring to 
light any deficiencies which could then be addressed by future refinements. 
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